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METHOD AND APPARATUS TO PROVIDE INTERACTIVE IMAGE CORRECTION FOR 
HEAVILY DAMAGED IMAGES 

CROSS REFERENCE TO RELATED APPLICATIONS 

This application is related to U.S. Patent Application Serial No. 
60/173,020, TITLED "Method to Correct Defects in Digital Images by Healing 
Across Defective Pixels," and U.S. Patent Appiication*.Serial No. 60/173,035, 
titled "Method and Apparatus for Correcting Large Defects in Digital Images by 
Stretching Across Defective Pixels," filed December 23, 1999, and which are 
hereby incorporated by reference in their entirety as if fully set forth herein. 

FIELD OF THE INVENTION 

The present invention relates to image processing and, particularly, to an 
improved system and method for correcting defects in images. 

DESCRIPTION OF THE RELATED ART 

Digitized images often include imperfections that are not present in the 
original image. One cause of such defects may be the components of the image 
capturing system. For example, a scanning surface, or platen, may contain 
scratches and other optical path obstructions. These optical path obstructions are 
digitized along with the original image and appear as imperfections in the digitized 
image. 

Another cause of imperfections is defects within the physical medium of the 
image. For example, a photograph, film negative, or other physical medium may be 
scratched or deformed despite careful handling. In addition, dust, hair, smudges, 
and the like may be deposited on the surface of the physical medium and will be 
digitized along with the original image. These defects will also appear in the 
digitized image as defects. 

Conventional software programs are available to identify such defects. Such 
programs typically create "binary" defect maps that mark each pixel in the defect 
map as defective or non-defective. Other programs may also provide "continuous" 
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defect maps, wherein each pixel is marked with a defect value proportional to the 
severity of any defect in the pixel. 

Once a pixel has been marked as defective, conventional imaging > 
programs replace the defective pixel with a replacement amplitude value 
5 determined from amplitudes of the surrounding pixels within a predetermined 
area. This interpolated value is then used to "fill" the defective pixel. 

After such automatic image correcting systems have been run, there may 
remain certain defects which the system cannot correct. Such defects may, for 
example, be of such amplitude as to result from a hole in the film. One 

10 conventional method allows the user to reset the system such that more pixels 
are marked for correction. However, increasing the range of pixels that are 
marked for correction can cause the image to become softer, thereby losing 
much of the detail. Another conventional method of dealing with such defects is 
to allow the user to manually identify and manually fill in the remaining defective 

15 pixels. However, visual inspection can be unreliable, and defective pixels can be 
overlooked. 

Such conventional methods are further disadvantageous in that the fill 
pattern and color is derived from the surrounding pixels, including those which 
have been marked as defective. Consequently, the corrected pixel may not 
20 appear true to the image. 

Conventional defect correction programs thus have several technical 
disadvantages when applied to heavily damaged images. For example, certain 
defect correction programs can soften image detail. Other image correction 
programs rely on visual inspection, which can be unreliable. Finally, the use of 
25 defective pixels for correction can result in inaccurate image correction. 

SUMMARY OF THE INVENTION 

These and other drawbacks in the prior art are overcome in large part by 
a system and method for correcting defects in heavily damaged images 
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according to the present invention. In accordance with one implementation, a 
secondary defect map, defining heavily damaged portions of the image which 
were not corrected automatically, is generated. The secondary defect map 
allows the user to correct the defects by highlighting the uncorrected areas and 
applying a defect correction process to the uncorrected areas. 

Thus, the defect correcting system and method "provides an improved 
method for correcting heavily damaged images by generating a defect map of 
those defects that are so severe that an automatic defect correction system 
cannot correct them. The defect correcting system and method then provides 
the user with various options for correcting the defects. These include selection 
of a replacement value from a palette of values, interpolation between values, or 
stretching or healing across the defect region. 

BRIEF DESCRIPTION OF THE DRAWINGS 

A better understanding of the invention is obtained when the following 
detailed description is considered in conjunction with the following drawings, in 
which like reference numerals represent like features, in which: 

FIG. 1 is a block diagram of an imaging system employing an image 
correcting system in accordance with the present invention; 

FIG. 2A-2B are diagrams illustrating exemplary primary and secondary 
defect maps; 

FIG. 3 is a flowchart illustrating operation of an implementation of the 
present invention; 

FIG. 4 is a flowchart illustrating operation of an implementation of the 
present invention; 

FIG. 5 is a flowchart illustrating operation of an implementation of the 
present invention; 

FIG. 6 is a flowchart illustrating operation of an implementation of the 
present invention; 
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FIG. 7 is a flowchart illustrating operation of an implementation of the 
present invention; 

FIG. 8 is a flowchart illustrating operation of an implementation of the 
present invention; 

FIG. 9 is a flowchart illustrating operation of an implementation of the 
present invention; ^ 

FIG. 10 is a flowchart illustrating operation of an implementation of the 
present invention; and 

FIG. 1 1 is a flowchart illustrating operation of an implementation of the 
present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

FIGS. 1-1 1 illustrate an image defect correcting system and method. As 
will be discussed in greater detail below, an image defect correcting system 
according to the present invention uses a defect map of heavily defective regions 
of an image and allows a user to correct them by choosing a defect correction 
process. In one embodiment, the defect correcting process comprises 
determining a correction value for each pixel in a region from values of "good" 
pixels surrounding it. 

Turning now to the drawings and, with particular attention to FIG. 1, an 
exemplary computer system 10 including image processing functionality 
according to the present invention is shown. The computer system 10 may be 
embodied as a known computer system, such as an IBM RS 6000 workstation, 
an IBM PC or PC-compatible computer, or an Apple G4 computer. 

The computer system 10 implements an image processing system 12. 
such as a computer program stored in memory (not shown) and executable by 
one or more processors (not shown). The computer system 10 may further 
include an image input device 14. The image input device 14 may be embodied 
as a scanner or as any device suitable for providing an image to be processed. 
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The image processing system 12 includes an image correcting unit 16, 
such as a computer program, in accordance with the present invention. As will 
be discussed in greater detail below, the image correcting unit 16 uses a 
selection unit 18, including, for example, a defect map 20 for selecting a pixel for 
5 image correction. One method for generating a defect map is described in U.S. 
Patent No. 5,266,805, which is hereby incorporated by-reference in Its entirety as 
if fully set forth herein. This method transmits infrared light at one surface of a 
film. A corresponding digitized infrared image is captured on this opposite side. 
Imperfections are identified because they substantially reduce or totally occlude 

10 the infrared image. The digitized infrared image thus provides an accurate map 
of the location and shape of image defects. Imperfections may then be 
corrected by conventional automatic interpolation methods. 

However, certain defects are of sufficient magnitude that they cannot be 
corrected using automatic techniques. As such, according to an implementation 

15 of the present invention, a second defect map 22 defining heavily defective 

pixels is generated, with the heavily defective pixels highlighted. A user interface 
(not shown) then gives the user the option of correcting the heavily defective 
pixels. If the user chooses to do so, the heavily defective pixels are corrected 
using one of the user defined correction methods 19. The correction methods 19 

20 include pixel stretching 21, healing 23, one-to-one replacement 25, defect region 
correction 27, interpolation 29, palette selection 31, or cut-and-paste method 33. 
It is noted that other methods may be employed; the figure is exemplary only. 

Operation is illustrated more clearly with reference to FIG. 2A. FIG. 2A 
iliustrates'a defect map 200a, with each square representing a pixel. A pixel of 

25 interest 202, which is to be corrected, is denoted X. Defective pixels 204 are 
denoted d. For example, a pixel may be identified as being defective if its 
defectiveness value exceeds a predetermined threshold. The remaining pixels 
205 are non-defective. 

The defective pixels 202, 204 are relatively small and may be corrected 
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using a variety of image correcting schemes, such as Digital ICE, available from 
Applied Science Fiction, Inc. Another method is described in U.S. Provisional 
Patent Application Serial No. 60/158,501, titled "System and Method for 
Correcting Defects in Digital Images Using Selective Fill In from Surrounding 
Areas," filed October 8, 1999, which is hereby incorporated by reference in its 
entirety as if fully set forth herein. v 

However, certain pixels may be so heavily defective that automatic 
correction programs do not correct them. That is, their level of defectiveness 
exceeds a predetermined threshold. An image correcting system according to 
an implementation of the present invention generates a secondary defect map 
22 (FIG. 1) of such heavily defective pixels and allows a user to correct them. 
Such a defect map 200b is shown in FIG. 2B. Heavily defective pixels 206 are 
denoted h. 

As will be explained in greater detail below, a variety of methods may be 
used to generate the secondary defect map 200b. In one embodiment, the 
secondary defect map 200b is generated at the same time the primary defect 
map 200a is generated, such as using a pixel by pixel defectiveness threshold 
comparison. In another embodiment, the secondary defect map 200b is 
generated after a first defect correcting method has been performed on the 
image data. More particularly, after such a program automatically corrects pixels 
which can be corrected, a number of heavily damaged pixels may remain 
uncorrected. The image correcting program identifies these pixels and 
generates the secondary defect map therefrom. Once the secondary defect map 
has been generated, the user may correct heavily damaged pixels 206 by 
selecting a defect correction method 14. 

One such correction method is the stretching method 21 described in U.S. 
Patent Application Serial No. 60/173,035, titled "Method and Apparatus for 
Correcting Large Defects in Digital Images by Stretching Across Defective Pixels," 
filed December 23, 1999. Another such correction method is the healing method 23 


WO 01/48694 


PC17US00/35323 


7 

described in U.S. Patent Application Serial No. 60/173,020, titled "Method to Correct 
Defects in Digital Images by Healing Across Defective Pixels," filed December 23, 
1999. 

Another method is a replacement method 25, in which a value for the pixel 
208 is obtained using a value from an adjacent or nearby non-defective pixel. 
For example, in FIG. 2B, the value for pixel 21 1 may be used to replace the pixel 
208. 

According to a surrounding area method 27, a replacement value is 
obtained using only pixels from a surrounding area that are themselves non- 
defective. For example, as shown in FIG. 2B, the user may select pixel 208. 
The image correcting system identifies a correction region 210 surrounding the 
pixel 208. Then, the replacement value is obtained using the values of pixels 
within the non-defective region 210. For example, the values of the non- 
defective pixels 212 may be averaged to obtain the replacement value for the 
pixel 208. 

According to an interpolation method 29, a user may select a plurality of 
nearby pixels and have a replacement value obtained by an interpolation 
process. For example, in FIG. 2B, the user can select a pixel 21 1 and a pixel 
213. An interpolation is performed on the selected pixels and a replacement 
value is obtained. 

Using a palette method 31, a replacement value may be obtained using a 
palette of values (not shown). The user selects a value from the palette to 
replace the defective pixel. 

. Finally, using a cut-and-paste method 33, the user can select a region on 
the defect map, cut it, and move it to lie atop the defective region. For example, 
with reference to FIG. 2B, the user can select the region 217, cut it, and copy it 
over to overlie the defective pixels h of the defect region 210. 

Generation of the secondary defect map according to one implementation 
of the invention is illustrated with reference to FIG. 3. In a step 402, a primary 
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defect map 20 is generated by the selection unit 18. The primary defect map 20 
is the defect map that will be initially used by the image correcting unit. In a step 
404, the image correcting unit 16 compares defectiveness values for all the 
pixels in the defect map 20 to a threshold value. In a step 406, the image 

5 correcting unit 16 and selection unit 18 generate the secondary defect map 22 
using those pixels whose defectiveness was determined in step 404 to exceed 
the threshold value. 

An alternate method for determining the secondary defect map 22 
according to an implementation of the present invention is shown with reference 

10 to FIG. 4. In a step 502, the selection unit 18 generates a primary defect map 

20. In a step 504, the auto-correction process is run by the image correcting unit 
16. In a step 506, the image correcting unit 16 determines which of the defective 
pixels have not been corrected. In a step 508, the image correcting unit 16 and 
selection unit 18 generate the secondary defect map 22 using those pixels. 

15 Operation of an implementation of the present invention is illustrated with 

reference to the flowchart of FIG. 5. In a step 510, the image correcting unit 16 
and selection unit 1 8 correct the image using a first defect map 20. In a step 
504, the image correcting unit 16 and selection unit 18 display the secondary 
defect map 22, identifying pixels which were not corrected on the first defect map 

20 20. In a step 514, the user selects one of the correction methods 19 to use in 

correcting defects on the defect map 22. In a step 516, the image correcting unit 
16 uses the selected correction method 19 to correct the defects on the defect 
map 22. . 

For example, FIG. 6 illustrates pixel replacement using the stretching and 
25 healing methods 21, 23 described above. In a step 802, a defect region is 
selected on the secondary defect map 22. In a step 804, the user selected 
stretching method 21 or healing method 23 is used to correct the defect. 

FIG. 7 illustrates pixel replacement using the individual replacement 
method 25. In a step 702, the user selects a pixel on the secondary defect map 
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22 for replacement. In a step 704, the user selects a pixel whose value(s) will be 
used to replace the defective pixel. In a step 706, the defective pixel is replaced. 

FIG. 8 illustrates a surrounding area method 27. In a step 602, a user 
selects a pixel on the secondary defect map 22. In a step 604, the image 
correcting unit 16 and selection unit 18 define a correction region around the 
defective pixel. In a step 606, the defective pixel is corrected using values 
derived from the pixels in the correction region. 

FIG. 9 illustrates an interpolation method 29. In a step 902, a user selects 
a defective pixel in the secondary defect map 22 for correction. In a step 904, 
the user selects one or more other pixels to be used for correction. In a step 
906, an interpolation is performed on the selected pixels to obtain a single value. 
In a step 908, the value is used to replace the defective pixel. 

FIG. 10 illustrates a palette method 31. In a step 910, a user selects a 
pixel for correction in the secondary defect map 22. In a step 912, the image 
correcting unit generates a palette of replacement values. For example, the 
palette may show colors of pixels that are located near the defective pixel. In a 
step 914, the user selects a value from the palette. In a step 916, the value is 
used to replace the defective pixel. 

A cut-and-paste method 33 is shown in FIG. 11. In a step 920, a user 
selects a replacement region 217 (FIG. 2B). In a step 922, the user copies or 
cuts the region 217. In a step 924, the region is positioned to overlie a defective 
region. 

The invention described in the above detailed description is not intended 
to be limited to the specific form set forth herein, but is intended to cover such 
alternatives, modifications and equivalents as can reasonably be included within 
the spirit and scope of the appended claims. To aid the Patent Office and any 
readers of any patent issued on this application in interpreting the claims 
appended hereto, applicants wish to note that they do not intend any of the 
appended claims to invoke paragraph 6 of 35 U.S.C. Sec. 1 12 as it exists on the 
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date of filing hereof unless the words "means for" or "step for" are used in the 
particular claim. 
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CLAIMS 

WHAT IS CLAIMED IS: 

5 1. A method, comprising: 

providing an image correcting unit for correcting an image using a first 
defect map; 

providing a selection unit for displaying a second defect map of pixels that 
were not corrected using said first defect map; and 
10 providing a method for selecting one or more defective pixels in said 

second defect map for correction. 

2. A method according to claim 1, including providing a method for using 
values of nondefective pixels within a correction region surrounding said one or 

15 more defective pixels to obtain a replacement value for said one or more 
defective pixels. 

3. A method according to claim 2, said displaying a second defect map 
comprising comparing one or more pixels in said first defect map with a threshold 

20 value. 

4. A method according to claim 2, said displaying a second defect map 
comprising determining which pixels on said first defect map were not corrected 
using said first defect map. 

25 

5. A method for correcting an image, comprising: displaying a secondary 
defect map of pixels that were not corrected using a primary defect map. 

6. A method according to claim 5, including using values of nondefective 
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pixels within a correction region surrounding one or more defective pixels on said 
secondary defect map to obtain a replacement value for said one or more 
defective pixels. 

7. A method according to claim 5, said displaying a secondary defect map 
comprising comparing one or more pixels in said primary defect map with a 
threshold value. 

8. A method according to claim 5, said displaying a secondary defect map 
comprising determining which pixels on said primary defect map were not 
corrected using said primary defect map. 

9. A system for correcting an image, comprising: 
an image input device; 

an image correcting unit adapted to correct said image using a first defect 

map; 

a selection unit configured to display a second defect map of pixels that 
were not corrected using said first defect map; 

wherein said selection unit is further configured to select one or more 
defective pixels in said second defect map for correction. 

10. A system according to claim 9, said image correcting unit configured to 
use values of nondefective pixels within a correction region surrounding said one 
or more defective pixels to obtain a replacement value for said one or more 
defective pixels. 

11. A system according to claim 1 0, said selection unit adapted to compare 
one or more pixels in said first defect map with a threshold value. 


WO 01/48694 


PCT/US00/35323 


13 

12. A system according to claim 9 t said selection unit configured to determine 
which pixels on said first defect map were not corrected using said first defect 
map. 

13. A systerri comprising: 

a selection unit adapted to display a secondary defect map of pixels that 
were not corrected using a primary defect map; and 

a user interface operable to receive instructions from a user for choosing 
a user correction method for correcting the pixels of the secondary defect map. 

14. A system according to claim 13, wherein the user correction method 
comprises using values of nondefective pixels within a correction region 
surrounding one or more defective pixels on said secondary defect map to obtain 
a replacement value for said one or more defective pixels. 

15. A system according to claim 14, said selection unit adapted to compare 
one or more pixels in said primary defect map with a threshold value. 

16. A system according to claim 14, said selection unit adapted to determine 
which pixels on said primary defect map were not corrected using said primary 
defect map. 
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